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ABSTRACT 

A critical linkage in training systems is the 
translation of training evaluation information for reassessing 
training needs and for making training program changes. This paper 
presents the development of a conceptual framework for examining the 
job relevancy and efficiency of training and the linkage for this 
evaluative information to training needs reassessment. How to 
integrate job performance information into the existing training 
evaluation system for identifying over- and undertrainer* tasks is 
also described. The Aerospace Ground Equipment (AGE) (AFS423X5) 
Technical Training Program is used as a case analysis for this 
report. Recommendations for future research to identify the content 
domain of an Airman Basic-in-Residence (ABR) Training program and to 
integrate performance information into the evaluation system are 
presented. (Author) 
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SUMMARY 



A critical linkage in training systems is the translation of 
training evaluation information for reassessing training 
needs and for making training program changes. This report 
presents the deveiopinent of a conceptual framework for 
examining the job relevancy and efficiency of training and 
the linkage of this evaluative information to training needs 
reassessment. How to integrate job performance information 
into the existing training evaluation system for identifying 
over- and undert rained tasks is also described. The 
Aerospace Ground Equipment (AGE) (AFS423X5) Technical 
Training Program is used as a case analysis for this report. 
Recommendations for future research to identify the content 
domain of an Airman Basic-in-Residence (ABR) Training 
program and to integrate performance information into the 
evaluation system is presented. 
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LINKING TRAINING EVALUATION TO TRAINING NEEDS ASSESSMENT: 
DEVELOPMENT OF A CONCEPTUAL MODEL 



I • INTRODUCTION 

In fiscal year 1989, over 35,000 airmen received 
initial technical skills training at resident Air Training 
Command (AtC) schools to prepare them for first job 
assignments in the Air Force. This represents an enormous 
amount of resources devoted to initial technical training. 
Training that is irrelevant or inappropriate at the 
technical training level is therefore quite costly to the 
Air Force (Ruck, Thompson, & Stacy, 1987) . Active steps 
must be taken to ensure that technical training programs are 
job-related and efficient. 

The Instructional System Development Process 

Each initial technical training program is planned, 
developed and conducted through a systematic and complex 
five step process. This Instructional System Development 
(ISD) process which guides the development and revision of 
technical training programs includes the following steps: 
(a) analyze system requirements; (b) define education and 
training requirements; (e) develop objectives and tests; (d) 
plan, develop, and validate instruction; and (e) conduct and 
evaluate instruction (AF Manual 50-2, 1979). 

The fifth step in the ISD process specifies that a 
thorough evaluation of the training program must be 
conducted. The linkage between the evaluation phase (step 
5) and the analysis of system requirements or training needs 
assessment (step 1) implies that training is a continuously 
evolving process that uses evaluative information to adapt 
and to improve its quality. Therefore, a critical linkage in 
the ISD process is the translation of training evaluation 
information into input useful for reassessing training needs 
and for making meaningful training program changes. 

Objective of Project 

As part of the ISD process, ATC has developed a formal 
training evaluation system to allow for the orderly 
development of change in the technical training programs. 
Given that the evaluation system was d3veloped over 10 years 
ago, it would seem reasonable to examine and re-evaluate the 
comprehensiveness and effectiveness of the evaluation 
system. This examination focuses on whether or not the 
evaluation system is limited in it;^ ability to succinctly 
and reliably answer questions about the quality of technical 



training programs or to provide the type of information 
required for use by training managers in making course 
revisions and continuous improvements. 

To critically examine the training evaluation system, a 
conceptual framework must be developed to guide the effort. 
The objectives of this project are to: (a) develop 
conceptual frameworks for understanding the components of a 
comprehensive training evaluation system, (b) describe the 
training evaluation system used in ATC training programs 
through a case analysis of a specific technical training 
program, (c) critically examine the training evaluation 
system in relation to the conceptual models developed, and 
(d) present recommendations for research to examine the 
complex issues of better linking training evaluation 
information to training needs reassessment. 



II. A CONCEPTUAL FRAMEWORK FOR TRAINING EVALUATION 

A comprehensive analysis of training needs is critical 
to training program development. Similarly, an initial 
assessment of training needs should also provide information 
for the development of a comprehensive training evaluation 
system. The key question asked during this evaluation 
development phase is "what do we want to know about the 
training program that will indicate the quality of the 
program?" In this regard, decision makers may be interested 
in answering one or more evaluation questions ice,, 
evaluation may be required for different purposes. To 
answer each evaluation question or to fulfill each purpose, 
different types of information or data must be gathered by 
an evaluation system. The conceptual framework relating 
training evaluation's purpose and types of evaluative 
information needed is presented below. 

A Typology of Training Evaluation Purposes 

The purpose of the evaluation drives the kinds of 
information that needs to be collected. Thus, if the 
purpose (s) behind the development of an evaluation system 
for a particular training program is not clearly specified 
during the initial analysis or needs assessment phase, it is 
unlikely that the appropriate information will be collected 
to link subsequent evaluation data with training needs 
reassessment. Without a clear sense of purpose, 
modifications to enhance the quality of the training program 
must rely on the intuition of the people who develop the 
training program (Montague & Wulfeck, 1986). 

There are five major purposes for conducting training 
evaluation. Table 1 presents the five purposes, the 
questions asked for each purpose and the types of 
information needed to answer the questions posed by each. 
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One purpose for conducting training evaluation is to 
determine the content validity of the training program 
(Goldstein, 1986) . Content validity asks the question "is 
the training content job relevant?." To answer this 
question, information about the tasks being performed on the 
job must be matched with the tasks being trained in the 
training program. A content validity ratio can be 
calculated to determine the extent to which the content of 
the training program is job relevant (Ford & Wroten, 1984). 

A second training evaluation purpose is to examine 
training efficiency. Training efficiency asks the question 
"is the training program over- or undertraining certain 
tasks (or Knowledges, Skills and Abilities (KSAs))?." To 
answer this question, information regarding the importance 
of tasks in the job domain and the emphasis of those tasks 
in the training program must be acquired. A matching 
technique can then be applied to examine areas of over- or 
undertraining (Ford & Wroten, 1984). Ore outcome of this 
analysis could be the elimination of certain tasks from the 
technical school training program that might be better 
trained on the job. 

A third purpose for conducting training evaluation is 
to determine training validity. Goldstein (1986) states 
that this purpose asks the question "did the trainees learn 
the material that was being trained?." To answer this 
evaluation question, one must collect information about the 
performance of the trainees during and/or at the end of the 
training program. The analysis determines the extent to 
which learning has taken place in comparison to a specified 
standard or criterion of success. 

A fourth purpose of evaluation is to determine transfer 
validity (Wexley, 1984). Transfer validity asks the 
question "are people performing well on the job after 
training?." To answer this question, information regarding 
performance on the job must be collected. Job performance 
must then be compared to some criterion of success to 
determine the extent of successful transfer. Experimental 
designs can be used to determine if the transfer of 
knowledges and skills have actually occurred and whether the 
change is attributable to the training program. 

A fifth purpose of training evaluation is to determine 
the predictive validity of a training program. At times, 
training is used as a device to select or place individuals 
into a particular position or job. Therefore, the question 
that must be answered with this evaluation purpose is "does 
training performance predict job performance so that 
accurate selection/placement decisions can be made?." The 
information that must be collected to conduct this analysis 
includes training and job performance information. 



Types of Training Evaluation Information 



Two major dimensions of information available for 
answering the questions posited for each evixluation purpose 
are represented in Table 1. One dimension is the source of 
the information that is gathered. The two sources of 
information available for a comprehensive training 
evaluation system are from the job domain and the training 
domain. A second dimension is the type of information 
gathered. The type of information is conceptualized as 
being task-based or performance-based. Task-based 
information focuses on what an individual "does do" while 
performance information examines "how well an individual 
does the specified tasks." 

Figure 1 presents a conceptual representation of the 
four domains that result from combining the two types of 
sources with the two types of information available. The 
four domains are the: (a) job content domain, (b) training 
content domain, (c) job performance domain, and the (d) 
training performance domain. 

The job and training content domains are conceptualized 
as consisting of a task and an emphasis component. The 
"task" component of the job content domain involves the 
identification of what tasks (or KSAs) are performed on the 
job. The "task" component of the training content domain 
includes information regarding what tasks (or KSAs) are 
taught during training. 

The "emphasis" component of the job content domain 
involves an analysis of how "important" the various tasks 
are for job performance. Importance can be operationalized 
in a number of ways including "importance to job 
performance" and "task difficulty." The "emphasis" 
component of the training content domain concerns the amount 
of effort devoted to training the various tasks included in 
the training program. Emphasis can be operationalized as a 
subjective assessment of effort devoted in training to each 
task or as the actual amount of time spent training each 
task. 

The job and training performance domains are 
conceptualized as consisting of a knowledge and a 
performance component. The "knowledge" component of the job 
performance domain focuses on how well the individual knows 
the appropriate ways to perform job tasks. Job knowledge 
tests are often used to address this issue. The 
"performance" component specifies how well the individual 
actually performs the job tasks. Information from job 



performance ratings, work samples and job indices such as 
quantity or quality of work can be gathered to determine the 
level of job performance. 

For the training performance domain, the "knowledge" 
component examines how well the trainees have learned the 
material that has been taught, Paper-and-pencil tests on 
the material covered in the course are typically used to 
examine the knowledge component. The "performance" 
component focuses on how well the trainees can perform the 
tasks that have been trained. Performance ratings, work 
samples, and other objective indices can be used to 
determine performance level while the trainees are 
progressing through the training program. 

Sammarv 

In summary, the model in Figure 1 provides a conceptual 
framework for describing the information available to answer 
training evaluation questions- The specific information 
gathered is dependent upon the purpose (s) for conducting 
training evaluation (see Table 1) . An examination of Figure 
1 also indicates that a critical boundary condition 
surrounding these four domains is information quality. This 
means that the usefulness of any information collected for 
answering the evaluation question posed is heavily dependent 
upon the quality of that inf ozonation. 

III. ANALYSIS OF THE AF ABR TRAINING EVALUATION SYSTEM 

The conceptual models developed in this paper highlight 
the important role evaluation systems have for enhancing the 
quality of training. This section describes the existing 
evaluation system relevant to the four domains of 
information presented in Figure 1. The Aerospace Ground 
Equipment (AGE) (AFS423X5) Airmen Basic-in-Residence (ABR) 
course at Chanute AFB, Illinois is used as a case study. 

Training Evaluation Information 

There are many documents in the Air Force that provide 
information that is useful in defining the job and training 
content and performance domains (see Appendix A for 
listing) . Table 2 lists the four domains along with the 
sources of information available in the Air Force training 
evaluation system, the type of information they contain, and 
the level of specificity of the various types of evaluative 
information. 

Level of specificity refers to the amount of detail in 
the evaluative information collected. The Air Force uses a 
task- based approach to needs assessment and training 
evaluation. An examination of this evaluation system reveals 
three levels of specificity in the task-based data that is 
collected. The basic component of the evaluation system is 



Table 2. The Available Sources of Evaluation Information. 



Domain 



Source of 
Information 



Type of 
Information 



Level of 
Specificity 



Job Content 



OSR 



STS 



U&TW 



Task 
Emphasis 

Task 

Knowledge 
Task 



T 
T 

CT 
CT 

CT & T 



Training 
Content 



POI 



Task 



CT 



Job 

Performance 



QA Checks 
TER 

TQR 



Performance 
Task 

Performance 
Task 

Performance 



T & TOSL 

CT 
CT 

CT 
CT 



Training 
Performance 



Block 
Tests 

Performance 

Progress 

Checks 



Knowledge 



Performance 



TOSL 



T => Task, CT => Cluster of Tasks, TOSL => Technical Order 
Step 
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at the "task" (labeled T in Table 2) level. A task is 
defined as a unit of work activity or operation which forms 
a significant part of a duty (AF Manual 50- 2, 1979), An 
example of task level information is found in the 
Occupational Survey Report (OSR) . 

An OSR, a product of 'he Occupational Measurement Center 
(CMC) , defines the job content of a career field and is 
based upon a comprehensive survey of airmen within a target 
occupation. The job inventory asks the respondents to 
indicate the tasks they currently perform. In the OSR, this 
information is recorded in one of three time-in-service 
categories, first enlistment (1-48 months) , second 
enlistment (49-96 months) , and career personnel (97+ 
months) . Tasks are analyzed to reflect the percentage of 
airmen performing each task in each category. These data 
provide task information about the job content domain by 
time-'in-service. Additional data collected include the 
relative time spent performing each task, the difficulty of 
learning each task and the emphasis that should be placed on 
training each task. 

Tasks can also be grouped together in terms of co- 
performance. An example of tasks grouped into more general 
statements is found in the Specialty Training Standards 
(STS) (AF Regulation 8-13 & ATC Supplement, 1, 1987) that 
incorporate multiple OSR task statements into each standard. 
We have labeled this grouping as the "cluster of tasks" (CT) 
level in Table 2. 

Freguently, tasks are broken down into requisite steps 
needed to perform a given task. An example is the steps 
found in Technical Orders (TO) that are the individual 
procedures required to perform each OSR task. The TOs 
provide step by step instructions for conducting all Air 
Force equipment maintenance. This includes maintenance 
provided for aircraft, electronic systems, support vehicles, 
and aerospace ground support equipment. In Table 2, this 
level of task specificity is labeled as Technical Order Step 
Level (TOSL) . 

Another way to examine the subcomponents of a task or 
cluster of tasks is to identify their underlying job 
elements (Primoff , 1975) . This approach specifies the 
knowledge, skills and/or abilities (KSAs) needed to perform 
each task or cluster of tasks. An example of a job element 
approach for the cluster of tasks level is found in the STS. 
The STS not only lists tasks clustered by co-performance, 
but also identifies the knowledge required to perform duties 
at the 3-, 5-, and 7-levels of proficiency. An example of a 
job element approach for the task level of specificity is 
found in the Training Requirements Analysis (TRA) which 
accompanies each new OSR. The TRA, like the OSR, is a 
product of OMC. In the TRA, the underlying KSAs needed to 
perform each OSR task are identified. This provides 



Training Development Branch personnel (TDB) , who are 
responsible for designing technical training courses, with 
more specific information needed to refine course content. 

The following is a detailed description of the level of 
specificity for the evaluative information gathered for each 
of the four domains described in Figure 1. While the 
information collected is similar across career fields, the 
AGE career field is used to illustrate the issues relevant 
to training evaluation. 

The Aerospace Ground Ecfuipment Career Field 

The AGE career field maintains both powered and non- 
powered eqxiipment used to support aircraft in the Air Force. 
The equipment varies from small jet engines used to start 
jet aircraft, to tow bars used by ground personnel when 
moving aircraft. The ABR course for AGE lasts 18 weeks. 
After completing this course, airmen are sent to an 
operational base and support a wide diversity of Air Force 
missions . 

Consequently, the ABR course is required to teach a 
variety of different tasks that any one airman might 
encounter in his or her first job assignment. 

Job Content Domain 

The most recent OSR for AGE surveyed 2,629 airmen in 
1983. About 75 senior AGE Noncommissioned Officers (NCOs) 
from different Major Commands (MAJCOM) completed two 
additional surveys. The purpose of these latter surveys was 
to gather "task difficulty" and "training emphasis" ratings 
information. Task difficulty is defined as the relative 
difficulty to learn a task and is rated on a nine point 
scale from 1 (extremely low) to 9 (extremely high) . 
Relative difficulty represents the time needed, on the 
average, to learn to perform a task satisfactorily. 
Training emphasis is defined as the perceptions of the NCOs 
as to which tasks should be taught to first enlistment 
airmen in a structured training course. This scale ranges 
from 0 (no training recommended) to 9 (extremely high 
training emphasis) . in relation to the conceptual framework 
presented in Figure 1, these two pieces of data are part of 
the emphasis component of the job content domain. 

In addition to the OSR data. Specialty Training 
Standards (STS) are created (current STS 42 3X5, dated Aug 
1987; Appendix B provides a portion of the current STS for 
AGE) . The 1983 OSR had over 600 task items that defined 
the AGE career field. The AGE STS dated July 1982 consisted 
of 124 statements, of which 80 were statements at the CT 
level of specificity. The remaining 44 STS statements were 
general knowledge statements. 
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Finally, the job content domain also includes Technical 
Orders for the AGE career field that are used whenever a 
task is performed. The TOs include the equipment needed as 
well as all of the steps needed to correctly perform the 
tasks relevant to the AGE career field. 

Training Content Domain 

The job content domain is defined at three different 
levels of specificity; the CT level is defined by the STS; 
the task level is defined by the OSR; and the step level is 
defined by the TO, A TRA for the AGE career field has not 
yet been completed. 

The training content domain is defined by the Plan of 
Instruction (POI) • The POI is derived from a complex system 
of translating OSR and STS information into training 
objectives. This system is described below. 

The OSR task information is the criterion standard used 
when making decisions about which tasks to include in an ABR 
training course,. If the OSR reports that 30% of the airmen 
in their first enlistment perform a task then the task is 
considered for possible inclusion in the course. Also taken 
into consideration by the AGE TDB is the task difficulty and 
training emphasis ratings. 

After an OSR is completed , personnel from OMC contact 
Subject Matter Experts (SMEs) to collect additional 
information to help define the training domain. The SMEs go 
through the OSR tasks on^'-by-one and determine within which, 
if any^ STS statement the task belongs. In some cases, a 
number of OSR tasks are matched to one STS statement. For 
example, 50 OSR tasks fall within a single AGE STS 
statement: 18A. "operational fundamentals of gas turbine 
compressors." Some STS statements are knowledge, not task 
statements, and so do not match with OSR tasks. 

As a next step, a Utilization and Training Workshop 
(U&TW) is convened to review the ABR training program and to 
make recommendations for content changes. The U&TW is 
composed of members from the MAJCOMS, ATC, and TDB. The STS 
is modified to reflect these recommended changes. This is a 
critical step as the STS indicates which statements are ABR 
course requirements and the level of proficiency the 
graduate is expected to obtain as a result of the training. 
The STS is a contract between the MAJCOMs and the AGE 
schoc". . It sets the criteria for knowledge and performance 
of the ABR graduates. The statements are written in general 
terms (CT level) in order to be as useful to as many users 
as possible (e.g. AGE STS statement #11. Reciprocating 
Engines, b. Diesel Engines, (2) inspect.). 

After the U&TW is corapleted, the TDB examines the new 
STS and other relevant material to identify the STS 
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statements that apply to the ABR course. OSR tasks that are 
listed under the applicable STS statement are reviewed in 
the following manner. If a task is performed by over 50% of 
the airmen during their first enlistment that task is 
recommended for knowledge and hands on training. If the 
percentage performing is between 30 - 49% then the task is 
recommended for knowledge training only. Below 30% tasks 
are generally not taught in the ABR course. The task 
difficulty and training emphasis ratings are also 
considered. A task is not recommended for training, no 
matter how high the percent performing rating is, if the 
task difficulty rating is statistically one standard 
deviation below the mean. A task is recommended for 
training if the training emphasis rating is statistically 
one standard deviation above the mean. The final criteria 
is the AGE school's ability to teach the task given the 
constraints the school faces with reference to time, 
equipment, qualified instructors, and building space. 

SMEs are also used to match the OSR tasks to the Plan 
of Instruction (PCI) objectives created by the AGE Training 
Development Branch (TDB) . This is done to insure that the 
tasks identified as needing to be trained through the OSR 
process are actually trained in the ABR course. Once this 
matching of tasks to objectives is accomplished, this 
matching information is not retained. Thus, while the POX 
references STS statements for each objective, there is no 
information on which OSR tasks underlying each STS are 
actually incorporated into the training content. This is 
important when changes are made in training content (e.g., a 
new OSR is completed or a U&TW inititates course changes) as 
there is no linkage between OSR task information and 
training objectives. 

The TDB personnel take the STS and the TOs in order to 
rewrite training objectives for the course. The TDB then 
develop criteria (e.g., similar equipment) for combining the 
objectives into blocks of instruction. Each block has one 
or more objectives and includes the time spent training on 
the objective, the STS statement reference, STS proficiency 
level, and the form of criteria testing, either written or 
progress check. For AGE there are 11 blocks of instruction 
which are further combined into five general instructional 
areas. 

Training Performance Domain 

The proficiency level of the trainers is examined 
through paper-and-pencil knowledge tests and hands-on 
performance tests. In the AGE ABR course trainees are given 
knowledge tests at the end of each of the 11 blocks. The 
questions are written by the course instructors and are 
directly related to the course objectives. A major criteria 
for determining a good test question is when less than 50% 
of the trainees get the question wrong. The average grade 
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on any given test is between 85% and 90%. Test questions 
are written at the sub-task level of specificity. As 
questions are written from steps in the TO. 

Many objectives in the ABR course require a performance 
progress check. The progress check involves two or more 
trainees working together to perform a task on AGE equipment 
located at the technical school. The trainees use the 
relevant TO to complete the task and are observed at all 
times by an instructor. If no difficulties are encountered 
during the performance progress check the trainees are 
checked off as having completed the task. If the trainees 
do not pass, the instructor explains what was done wrong and 
trainees are allowed to perform the task again. 

Job Performance Domain 

Training Evaluation Reports (TERs) are produced 
annually or bi-annually by Training Evaluation Division 
personnel at each technical training command. The TER 
examines how well the ABR course is preparing graduates to 
perform their assigned tasks according to the proficiency 
level specified by the STS. Surveys are developed to 
collect this information with questionnaire items taken 
directly from the applicable STS statements. 

For 1988, TER surveys were sent to 314 supervisors of 
recent AGE graduates asking them how satisfied they were 
with the graduate's performance and if the tasks taught in 
the course were being utilized. If less than 80% of the 
graduates perform the required tasks satisfactorily then 
training is considered inadequate. If less than 30% of the 
supervisors indicate the graduates are performing a task 
taught in training, consideration is given for removal of 
that task from the ABR course. A TER was recently completed 
(Feb 88) for the AGE ABR course. No recommended changes to 
course content were made based on this information. 

Supervisors in the field have an additional method of 
providing technical schools with feedback about training 
effectiveness in the form of Training Quality Reports (TQR) . 
TQRs are submitted by a supervisor and report a graduate's 
lack of proficiency in one or more tasks or the lack of 
knowledge that the STS states a graduate of the AGE ABR 
course should have. The school is required to respond to 
all TQRs. Between January 1985 and August 1988 7 TQRs have 
been submitted about the AGE ABR course. 

In addition, performance checks, similar to the 
progress checks performed in the technical school, are 
conducted by the Quality Assurance (QA) branch personnel in 
field settings. On a regular basis, QA monitors airmen 
performing different tasks to check on their proficiency in 
those tasks. These are tasks they are used to certify 
proficiency. QA personnel insure that the tasks are 
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completed in accordance with the procedures laid out in the 
TOs and that there are no major discrepancies. If a m^jor 
discrepancy is noted then recommendations are made which 
usually include decertification and/or additional training. 
This information is used by the MAJCOMs to monitor wing 
readiness, but is not presently used to provide performance 
feedback information to training programs. 

IV. CRITIQUE OF THE TRAINING EVALUATION SYSTEM 

Table 1 presented five purposes for conducting training 
evaluation. A critique of the existing training evaluation 
system in terms of these five purposes is clearly beyond the 
scope of this report. Instead, we have chosen to focus 
attention on the issues of content validity and training 
efficiency as they form the initial steps for answering 
questions regarding the three additional evaluation 
purposes. 

Content Validity Issues 

One of the purposes of an evaluation system is to 
insure that the training content is job relevant or content 
valid. (**is the training content job relevant?"). The 
information needed to address this question includes an 
analysis of the "task" components of the job and training 
content domains. Thus, the question of content validity 
must be answered through a direct comparison of the tasks 
(or knowledges and skills) being performed on the job with 
the tasks being taught in the training program. The greater 
the overlap between the tasks performed on the job and the 
tasks taught in the training program, the greater the 
content validity of the training. Tasks in the training 
program that are not important on the job are considered 
contaminants, while important tasks not included in the 
training program are considered training deficiencies. 

Therefore, to determine the content validity of a 
training program, the content of the job in terms of tasks 
performed and the content of the training program must be 
identified. Currently, the job content domain is specified 
at the task level by the OSR. The ABR training content 
domain is most clearly identified at the task cluster level 
by the STS. 

Therefore, it is difficult to determine training 
content validity at the task level of specificity. To more 
comprehensively evaluate content validity, the content of 
the training program must be clearly identified at the task 
level. Without direct information about the training content 
domain at the task level, there is no simple, systematic way 
of incorporating the new OSR information into the evaluation 
system to make meaningful changes in training content and 
training design. Instead, instructor knowledge of course 
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content is required to determine if an OSR task is currently 
taught in the AGE ABR course. Given duty rotations, it is 
highly likely that the instructors who helped make linkages 
between the old OSR and the POI are not available when the 
new OSR data is created. Thus, linkage information may be 
lost to the evaluation system. 

Training Efficiency Issues 

A content valid training program indicates that the 
tasks being performed on the job are the same tasks being 
trained. Once content validity is established, a second 
issue is training efficiency. Training efficiency asks the 
question "is the training program over- or undertraining 
certain tasks or KSAs?." Thus, a training program may have a 
high degree of job relevancy (training the appropriate 
tasks) but may or may not be placing an appropriate amount 
of emphasis on the various tasks during training to match 
their "need" for training. The information needed to 
address the training efficiency question includes an 
analysis of the "emphasis" components of the job and the 
training content domains (see Figure 1) . The question of 
efficiency can be addressed through a direct comparison or 
matching of the emphasis placed on the tasks during training 
with training "needs" i.e., how important the task is in the 
job domain. 

Ford and Wroten (1984) developed a methodology called 
the Matching Technique to link the "emphasis" components of 
the job and training content domains to determine training 
efficiency. The Matching Technique is conceptualized in 
Figure 2 as a matrix in which training emphasis is directly 
compared to training needs" (i.e., how important the task is 
for job performance) • 

The comparison of emphasis with needs identifies 
training "hits" and "misses." Training hits refer to those 
tasks where the emphasis received in training appropriately 
reflects training needs. Training misses can involve areas 
of deficiencies (undertraining) and excesses (overtraining) . 
Training deficiencies are content areas whose high training 
needs are not matched by a high degree of emphasis in the 
training program. Training excesses are tasks that are 
receiving an excessive amount of emphasis relative to their 
need to be training. The greater the number of misses (both 
deficiencies and excesses) uncovered, the less efficient the 
training program is and the greater need for reassessment 
and redesign of the program. 

Any measure of training "needs" or training "emphasis" 
can be used to apply the Matching Technique as long as the 
measures reliably quantify the extent to which areas need to 
be trained or emphasized in the training program. 
Nevertheless, as noted when discussing content validity, the 
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comparison of training "needs" (i.e., job emphasis or 
importance) and training emphasis can only be as good as the 
quality of the data collected. Also the type of 
information (tasks, KSAs) collected on the job and training 
content domains must be operational ized at the same level of 
specificity to make this direct comparison possible. 

An examination of the evaluation information collected 
for the AGE ABR course reveals that the Matching Technique 
can not be completed as some of the information needed is 
not available. The training emphasis rating found in the OSR 
is the supervisor's perception of which OSR tasks should be 
taught in the AGE ABR course. The problem is that the 
training content domain has no comparable training emphasis 
information. Instroictor ratings of the emphasis placed on 
tasks during training or the actual time spent in training 
on tasks are two ways of collecting emphasis information in 
the training content domain. 

Another factor limiting the ability of the training 
system to adequately address the training efficiency issue 
is the lack of adequate training and job performance 
information. Although performance infoinnation is available 
in both the job and training domains, the information is 
neither used nor collected in a form that makes it useful to 
the training evaluation system. Performance information 
would be quite useful in determining over- or undertraining 
of tasks in the AGE ABR course. This issue is addressed 
further in the future research directions section below. 

V. FUTURE RESEARCH DIRECTIONS 

Based on the conceptual models developed and the 
analysis of the Air Force ABR Training system, two research 
needs are identified: (a) to systematically identify the 
training content domain in terms of tasks and emphasis 
placed on those tasks, and (b) to incorporate job 
performance information into the training system to address 
the issue of training efficiency. 

Identification of Training Content Domain 

The job content domain is defined through the OSR 
process in terms of the tasks performed on the job and the 
emphasis that should be placed on training those tasks 
during the ABR course. The OSR process represents a large 
commitment of resources to develop systematic data on the 
job domain which can be used for aiding training personnel 
of the ABR courses in updating training to maintain a high 
quality program. 

A major problem identified in this review is that 
similar efforts to systematically identify the tasks taught 
and the emphasis placed on those tasks in the training 
content domain are lacking. Information about the ABR 
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course is available through the POIs and lesson guides but 
is not at the same level of specificity as the OSR data. In 
addition, no data are gathered on how much emphasis is 
actually placed on each task that is part of the training 
content domain. Without such information, decision makers 
can not link the job and training content domains to fully 
answer questions of training content validity and training 
efficiency. 

Consequently, one research effort should be devoted to 
developing a methodology for identifying the training 
content domain in terms of tasks and emphasis. Training 
Branch personnel, instructors, students and recent graduates 
are all potential sources of information regarding the 
training content domain. 

For example, TDB staff members from AGE can be used as 
SMEs and asked to independently examine the list of OSR task 
statements from the most recent OSR (i.e., 1983) and state 
whether the task is taught or not in the ABR course. 
Disagreements among SMEs can be resolved through consensus 
judgment to provide a preliminary list of OSR task 
statements that are taught in the ABR course. Given that 
the ABR course is divided into five training areas, the SMEs 
could then match the preliminary list of task statements to 
the area in which each task is taught (some tasks may be 
taught across a number of areas) . This will provide a task 
matrix for each of the five training areas in the course. 
Once the matrix is created, instructors can then be used to 
verify the task list. Multiple instructors are assigned to 
each area and each instructor leads a class of graduates 
(usually 12) through his/her training area. Thus, surveys 
can be conducted to determine whether there is substantial 
agreement among the various instructors that the tasks 
identified are actually taught in the training program. This 
two step process (TDB staff SME judgments and instructor 
survey responses) would result in a list of tasks taught by 
area at the OSR task level of specificity. It would also 
allow for an exairdnation of the extent to which instructors 
in the same area teach the same tasks. 

Once the training content domain has been identified at 
the OSR level, the second phase of the research would be to 
develop a methodology for collecting information on how much 
emphasis is actually placed on each task trained in the AGE 
ABR course. Emphasis information can be collected from two 
sources. First, information regarding the amount of time 
actually spent training each task identified in Phase 1 can 
be determined through a survey of the AGE instructors. The 
survey would ask each instructor to independently estimate 
the amount of time spent training each task in his/her 
training area i.e., to distribute the total time available 
for training in an area across the tasks being taught in 
that area. This would allow for an analysis of the extent 
to which different instructors in the same training area 



18 



place the same amount of emphasis on different tasks trained 
in that area. Second, ratings of "relative emphasis" given 
to each task trained can be collected through a survey of 
recent graduates of the AGE ABR course. This would allow 
for a comparison of the instructor estimates of training 
emphasis with estimates of recent graduates. 

Integrating Training Efficiency and Performance Information 

The identification of the training content domain in 
terms of OSR tasks and the development of a methodology for 
determining training emphasis are prerequisites for using 
the Matching Technique (Ford & Wroten, 1984). The Matching 
Technique provides preliminary information that certain 
tasks may be training excesses (i.e., overtrained) or 
training deficiencies (i.e., undertrained) . To more fully 
address the question of ' training efficiency (are tasks over- 
or undertrained?) , performance information must be 
incorporated into the process. 

Figure 3 presents a 2 X 2 matrix which combines 
information that is created from ^he Matching Technique with 
job performance information. An analysis of the Matching 
Technique identifies tasks that are excesses and 
deficiencies. In Figure 3, performance level is 
conceptualized at the group level of analysis i.e., how well 
the group of individuals trained are performing various 
tasks on the job. For purposes of this example, 
perforraance is dichotomized into the categories of "not 
perfoirming well" and "performing well". The four 

components of the matrix in Figure 3 derived by combining 
the outcomes of the Matching Technique with performance 
level information provide data relevant to identifying over- 
or undertrained tasks. As an example, let us assume that a 
number of tasks were identified through the Matching 
Technique as being deficient. The next step is to examine 
how well recent graduates are actually performing those 
tasks on the job. If the tasks are not being performed 
well, then the need for more training emphasis during the 
training program is indicated. If the tasks are being 
performed well, this indicates that training deficiency is 
not an important issue to consider. Such a result may 
suggest that formal training of the tasks is not necessary, 

The Matching Technique may also identify some tasks as 
training excesses. An examination of job performance may 
indicate that some tasks that are training excesses are 
performed well and others are not perfomned well (see Figure 
3). If a task is not performed well on the job, this 
indicates that there is a serious problem that is best 
addressed through other means such as on-the-job training. 
If a task is perfomned well, this suggests that 
consideration be given to examining whether training 
emphasis placed on the task could be reduced without 
detrimental effects on job performance. Currently, job 
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performance information is not used to make decisions 
regarding training efficiency. A new methodology called the 
Job Performance Measurement System (JPMS) developed by the 
AFHRL (Hedge & Teachout, 1986) provides the type and quality 
of job performance data needed to identify over- and 
undertrained tasks. 

The JPMS contains three types of information. First, 
participants complete "a hands-on" portion of the Walk- 
Through Performance Testing procedure in which participants 
actually perform certain tasks and are scored by trained 
observers on whether the tasks are performed correctly. A 
second portion of the Walk-Through Performance Test is an 
interview portion which asks participants to explain how 
they would go about completing certain tasks rather than 
actually performing those tasks. These responses are scored 
by trained interviewers. Both the hands on testing and the 
interview method are scored at the TO (Step) level (subtasks 
are performed correctly or not) which are then summed for 
each task to provide a score at the OSR (task) level. The 
third type of information is a Specialty Job Knowledge Test 
which covers a number of tasks in the career field. 
Participants must respond to a number of multiple choice 
questions derived from the task content domain at the task 
level of specificity. 

Given the development of the conceptual model, the 
next step is the development of the methodology to integrate 
JPM data into the training evaluation system to address 
issues of over- or undertraining. If the training content 
domain is specified at the task level of specificity, then 
the integration of performance data laust be at the task 
level also. Nevertheless, for tasks that are identified as 
over- or undertrained, job performance at the TOSL can then 
be examined to determine which procedural steps are most 
problematic. Research could examine the impact of feeding 
back job performance information to training personnel on 
ABR course changes. 
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Waalilngcoa DC 20330-SOOO 
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(For AFSCa 42335/55/75) 
August 1987 



AEROSPACE GROUND EQUIPMENT MECHANIC 
AND 

AEROSPACE GROUND EQUIPMENT TECHNICIAN 

I. The l«pl««ntacloa of the STS for cechalcal training provided by Air Training Comt»«nd la wUti 
th« claaa entering 871005 and graduating 880209, wmrwna is witn 

^- Purpose of this Specialty Training Stan dard (STS) . As preacrlb^jd In AFR 8-13, this STS- 

a. , Lists In column 1 of attachment 1 the moat common tasks, knowledges, and technical 
references <TR) necessary for alraen to perform dutlea at the 3-. 5-, and 7-skin level AFSC In th^ 
Aerospace Ground Equipment ladder of the Airman Aircraft Accesao;y Maintenance career liS? These 
ar. based on the analysis of the duties In AFR 39-1, effective 31 October 1986 

b. Provides OJT certification columns In attachment 1 to record completion of taak and 
knowledge training requirements. Certification I, accomplished as outUn.S In AFR 50-23 

rn y^^A ^""""^ cralnlnj^ requirements. Column 3A<1) of attachment I shows the proficiency 

PD?C^r;;i5rd: cr^beS'l^'ipR'so's ^ "'"^^ °^ ^"^"^"8 ^" C3ABR4233 OOO ' 

th. eat^bl^fhin , / """"^ ^" "1""" 3A<1). the first code U 

the established requirement for resident training on the task/knowledge. The second cod. Is the 

ah'L*. tl orJacfen'cv't b. ^ """h' '° constraint..' Column 3C<1) of att^chm^nt L 

On ,h^'r.h^''Tf " J"" qualification standard for on-the-job training «hen placed In AF Form 623. 
On-the-Job Training Record, and uaed according to AFR 50-23. For OiT? the tasks m column 1 are 

«"l."ta„c*. ar'^'r'? «° 'r'- Indlvldual'can perfoc the task ilthLt 

...l.tanc. and meet ocal requ rements for accuracy, timeliness, and correct u.e of procedures. 
Fcr/Akrfr^ 1 " ^ knowledge provided In the 5-sklU level CDC. See 

ttl/AFSC/CDC Hating maintained by the unit OJT manager for current CDC listings 

Syatei'(UApl)* ^n,M?rVr' ^?''r"'T°^ P"?"°^^°" ^" Welght.d Alr,Un Promotion 

Syate. (WAPS) Specialty Knowledge Teats (SKTs) are developed at the USAF Occupational Measurement 
Center by senior NCOs with extensive practical experience in their c.r.er field.. The tests sarple 
knowledge Of STS subject .atter areas Judged by test development tea- member, to be moat 

I^'Tfl r ^1°^'°" " ''i*'" l*-"'! °o "udy ref.r.hce. li.ted m 

AFP 39-8. Individual responsibilities are In Chapter 14 of AFR 35-8. 

rhn.-*;, J'^S*^'"' ^ provide, the electronic fundamental requirements for this specialty. Only 
tno.e iteM coded are required by this AFSC. ' 

?• Proficiency Code Key. Attachment 1 contains the proficiency code key used to indicate the 
level of training and knowledge provided by resident training Jd care.r development courses. 

4. Recommendattona. Report unsatisfactory performance of Individual course Graduates as 
prescribed In AFR 50-38. Report Inadequacies of this STS through channels to HQ ATCmO 
Reference specific STS paragraphs. aiv-/iiu, 
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